Systems and methods for three-dimensional interaction monitoring in an ems environment

ABSTRACT

A method for tracking interactions in an emergency response environment according to embodiments of the present invention includes receiving color images and depth information from within a field of view of a sensor array; maintaining an emergency encounter record; monitoring one or both of a position of an object and movement of the object in the emergency response environment based on the color images and depth information received by the sensor array; and recording an occurrence of a condition in the emergency encounter record, wherein the condition is based on the one or both of the position of the object and the movement of the object.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication Ser. No. 61/707,671, filed on Sep. 28, 2012, and of U.S.Provisional Patent Application Ser. No. 61/707,665, filed on Sep. 28,2012, both of which are incorporated herein by reference in theirentireties for all purposes.

TECHNICAL FIELD

Embodiments of the present invention relate generally to gesturerecognition and three-dimensional interaction tracking in an emergencymedical services environment.

BACKGROUND

In an emergency medical services (“EMS”) or first responder environment,caregivers must often focus more acutely on patient care in a shorteramount of time and with a greater number of uncertainties and variablesthan their counterparts in a hospital setting. Creating a record of theEMS caregiver's encounter with a patient, however, remains important.Manual input of information into patient charting systems (e.g. bytyping or by writing) can sometimes take valuable time and attentionaway from patient care, can be distracting, and can often beinaccurately recreated from memory after an EMS encounter.

SUMMARY

A method for tracking interactions in an emergency response environmentaccording to embodiments of the present invention includes receivingcolor images and depth information from within a field of view of asensor array; maintaining an emergency encounter record; monitoring oneor both of a position of an object and movement of the object in theemergency response environment based on the color images and depthinformation received by the sensor array; and recording an occurrence ofa condition in the emergency encounter record, wherein the condition isbased on the one or both of the position of the object and the movementof the object.

The method of paragraph [0004], wherein the object is a human, andwherein monitoring one or both of the position of the object andmovement of the object comprises monitoring one or both of the positionof the human and movement of an at least partial skeletal approximationof the human.

The method of any of paragraphs [0004] and [0005], wherein the human isa first object, wherein the condition comprises the at least partialskeletal approximation of the human coming within a certain distance ofa second object.

The method of any of paragraphs [0004] through [0006], wherein the humanis a first human, and wherein the second object is a second human.

The method of any of paragraphs [0004] through [0007], wherein thecondition comprises the first human touching the second human.

The method of any of paragraphs [0004] through [0008], wherein thesecond human is a patient being treated by the first human in theemergency response environment.

The method of any of paragraphs [0004] through [0009], wherein recordingthe occurrence of the condition comprises recording a time at which thecondition occurs.

The method of any of paragraphs [0004] through [0010], wherein recordingthe occurrence of the condition further comprises recording a type ofthe condition.

The method of any of paragraphs [0004] through [0011], wherein recordingthe occurrence of the condition further comprises recording as videofootage the color images received during the occurrence of thecondition.

The method of any of paragraphs [0004] through [0012], furthercomprising: receiving streaming clinical data about a patient, andcorrelating at least a portion of the streaming clinical data in theemergency encounter record with the occurrence of the condition.

The method of any of paragraphs [0004] through [0013], whereincorrelating the at least the portion of the streaming clinical datacomprises flagging the at least the portion of the streaming clinicaldata corresponding to a time of the occurrence of the condition.

A system for tracking interactions in an emergency response environmentaccording to embodiments of the present invention includes a sensorarray, wherein the sensor array is adapted to receive color images anddepth information in its field of view; a control system communicablycoupled to the sensor array, the control system configured to: maintainan emergency encounter record; monitor one or both of position andmovement of an object in the emergency response environment based on thecolor images and depth information received from the sensor array; andrecord an occurrence of a condition in the emergency encounter record,wherein the condition is based on the one or both of position andmovement of the object.

A method for inventory control in an emergency response environment,according to embodiments of the present invention, includes detectingthree-dimensional movement of a human body in the emergency responseenvironment with a sensor array, wherein the sensor array generatesvisual information and depth information about the emergency responseenvironment; detecting three-dimensional movement of an object in theemergency response environment; determining an occurrence of contactbetween the human body and the object; and recording an entry in anemergency encounter record based on the occurrence of the contact.

The method of paragraph [0016], wherein the object is a narcoticmedication stored in an enclosure in the emergency response environment,the method further comprising: determining, based on the detection ofthe three-dimensional movement of the human body and the object, anoccurrence of intersection of the human body with the enclosure; andrecording an entry in the emergency encounter record based on theoccurrence of the intersection.

The method of any of paragraphs [0016] and [0017], wherein the object isa narcotic medication stored in an enclosure in the emergency responseenvironment, the method further comprising: determining, based on thedetection of the three-dimensional movement of the object, an occurrenceof removal of the narcotic medication from the enclosure; and recordingan entry in the emergency encounter record based on the occurrence ofthe removal.

The method of any of paragraphs [0016] through [0018], furthercomprising updating an inventory database, based on the occurrence ofthe removal, to reflect that the narcotic medication has been used andneeds restocking.

While multiple embodiments are disclosed, still other embodiments of thepresent invention will become apparent to those skilled in the art fromthe following detailed description, which shows and describesillustrative embodiments of the invention. Accordingly, the drawings anddetailed description are to be regarded as illustrative in nature andnot restrictive.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an emergency response environment with a vehiclecontrol system communicably coupled to other devices, according toembodiments of the present invention.

FIG. 2 illustrates a computer system, according to embodiments of thepresent invention.

FIG. 3 illustrates an emergency response environment with a system thatmonitors three-dimensional interaction, according to embodiments of thepresent invention.

FIG. 4 illustrates a system including a vehicle control system and asensor array, according to embodiments of the present invention.

FIG. 5 illustrates a table listing various hand and finger gestures thatmay be recognized by the system of FIG. 4, according to embodiments ofthe present invention.

FIG. 6 illustrates a table listing various head and facial gestures thatmay be recognized by the system of FIG. 4, according to embodiments ofthe present invention.

FIG. 7 depicts a flow chart illustrating a method for monitoringthree-dimensional interaction in an emergency response environment,according to embodiments of the present invention.

FIG. 8 depicts a flow chart illustrating a method for monitoringthree-dimensional interaction of a caregiver with a patient in anemergency response environment, according to embodiments of the presentinvention.

FIG. 9 depicts a flow chart illustrating a method for monitoringthree-dimensional interaction in an emergency response environment forinventory control, according to embodiments of the present invention.

FIG. 10 depicts a flow chart illustrating a method for gesturerecognition in an emergency response environment, according toembodiments of the present invention.

While the invention is amenable to various modifications and alternativeforms, specific embodiments have been shown by way of example in thedrawings and are described in detail below. The intention, however, isnot to limit the invention to the particular embodiments described. Onthe contrary, the invention is intended to cover all modifications,equivalents, and alternatives falling within the scope of the inventionas defined by the appended claims.

DETAILED DESCRIPTION

As illustrated in FIG. 1, a system 100 according to embodiments of thepresent invention performs advanced data management, integration andpresentation of EMS data from multiple different devices. System 100includes a mobile environment 101, an enterprise environment 102, and anadministration environment 103. Devices within the various environments101, 102, 103 may be communicably coupled via a network 120, such as,for example, the Internet. System 100 is further described in PatentCooperation Treaty Application Publication No. WO 2011/011454, publishedon Jan. 27, 2011, which is incorporated herein by reference in itsentirety for all purposes.

As used herein, the phrase “communicably coupled” is used in itsbroadest sense to refer to any coupling whereby information may bepassed. Thus, for example, communicably coupled includes electricallycoupled by, for example, a wire; optically coupled by, for example, anoptical cable; and/or wirelessly coupled by, for example, a radiofrequency or other transmission media. “Communicably coupled” alsoincludes, for example, indirect coupling, such as through a network, ordirect coupling.

According to embodiments of the present invention, the mobileenvironment 101 is an ambulance or other EMS vehicle—for example avehicular mobile environment (VME). The mobile environment may also bethe local network of data entry devices as well as diagnostic andtherapeutic devices established at time of treatment of a patient orpatients in the field environment—the “At Scene Patient MobileEnvironment” (ASPME). The mobile environment may also be a combinationof one or more of VMEs and/or ASPMEs. The mobile environment may includea navigation device 110 used by the driver 112 to track the mobileenvironment's position 101, locate the mobile environment 101 and/or theemergency location, and locate the transport destination, according toembodiments of the present invention. The navigation device 110 mayinclude a Global Positioning System (“GPS”), for example. The navigationdevice 110 may also be configured to perform calculations about vehiclespeed, the travel time between locations, and estimated times ofarrival. According to embodiments of the present invention, thenavigation device 110 is located at the front of the ambulance to assistthe driver 112 in navigating the vehicle. The navigation device 110 maybe, for example, a RescueNet® Navigator onboard electronic datacommunication system available from ZOLL Data Systems of Broomfield,Colo.

As illustrated in FIG. 1, a patient monitoring device 106 and a patientcharting device 108 are also often used for patient care in the mobileenvironment 101, according to embodiments of the present invention. TheEMS technician 114 attaches the patient monitoring device 106 to thepatient 116 to monitor the patient 116. The patient monitoring device106 may be, for example, a defibrillator device with electrodes and/orsensors configured for attachment to the patient 116 to monitor heartrate and/or to generate electrocardiographs (“ECG's”), according toembodiments of the present invention. The patient monitoring device 106may also include sensors to detect or a processor to derive or calculateother patient conditions. For example, the patient monitoring device 106may monitor, detect, treat and/or derive or calculate blood pressure,temperature, respiration rate, blood oxygen level, end-tidal carbondioxide level, pulmonary function, blood glucose level, and/or weight,according to embodiments of the present invention. The patientmonitoring device 106 may be a Zoll E-Series® or X-Series defibrillatoravailable from Zoll Medical Corporation of Chelmsford, Mass., accordingto embodiments of the present invention. A patient monitoring device mayalso be a patient treatment device, or another kind of device thatincludes patient monitoring and/or patient treatment capabilities,according to embodiments of the present invention.

The patient charting device 108 is a device used by the EMS technician114 to generate records and/or notes about the patient's 116 conditionand/or treatments applied to the patient, according to embodiments ofthe present invention. For example, the patient charting device 108 maybe used to note a dosage of medicine given to the patient 116 at aparticular time. The patient charting device 108 and/or patientmonitoring device 106 may have a clock, which may be synchronized withan external time source such as a network or a satellite to prevent theEMS technician from having to manually enter a time of treatment orobservation (or having to attempt to estimate the time of treatment forcharting purposes long after the treatment was administered), accordingto embodiments of the present invention. The patient charting device 108may also be used to record biographic and/or demographic and/orhistorical information about a patient, for example the patient's name,identification number, height, weight, and/or medical history, accordingto embodiments of the present invention. According to embodiments of thepresent invention, the patient charting device 108 is a tablet PC, suchas for example the TabletPCR component of the RescueNet® ePCR Suiteavailable from Zoll Data Systems of Broomfield, Colo. According to someembodiments of the present invention, the patient charting device 108 isa wristband or smart-phone such as an Apple iPhone or iPad withinteractive data entry interface such as a touch screen or voicerecognition data entry that may be communicably connected to the VCS 104and tapped to indicate what was done with the patient 116 and when itwas done.

The navigation device 110, the charting device 108, and the monitoringdevice 106 are each separately very useful to the EMS drivers 112 andtechnicians 114 before, during, and after the patient transport. Avehicle control system (“VCS”) 104 receives, organizes, stores, anddisplays data from each device 108, 110, 112 to further enhance theusefulness of each device 108, 110, 112 and to make it much easier forthe EMS technician 114 to perform certain tasks that would normallyrequire the EMS technician 114 to divert visual and manual attention toeach device 108, 110, 112 separately, according to embodiments of thepresent invention. In other words, the VCS centralizes and organizesinformation that would normally be de-centralized and disorganized,according to embodiments of the present invention.

The VCS 104 is communicably coupled to the patient monitoring device106, the patient charting device 108, and the navigation device 110,according to embodiments of the present invention. The VCS 104 is alsocommunicably coupled to a storage medium 118. The VCS 104 may be atouch-screen, flat panel PC, and the storage medium 118 may be locatedwithin or external to the VCS 104, according to embodiments of thepresent invention. The VCS 104 may include a display template serving asa graphical user interface, which permits the user (e.g. EMS tech 114)to select different subsets and/or display modes of the informationgathered from and/or sent to devices 106, 108, 110, according toembodiments of the present invention.

Some embodiments of the present invention include various steps, some ofwhich may be performed by hardware components or may be embodied inmachine-executable instructions. These machine-executable instructionsmay be used to cause a general-purpose or a special-purpose processorprogrammed with the instructions to perform the steps. Alternatively,the steps may be performed by a combination of hardware, software,and/or firmware. In addition, some embodiments of the present inventionmay be performed or implemented, at least in part (e.g., one or moremodules), on one or more computer systems, mainframes (e.g., IBMmainframes such as the IBM zSeries, Unisys ClearPath Mainframes, HPIntegrity NonStop servers, NEC Express series, and others), orclient-server type systems. In addition, specific hardware aspects ofembodiments of the present invention may incorporate one or more ofthese systems, or portions thereof.

As such, FIG. 2 is an example of a computer system 200 with whichembodiments of the present invention may be utilized. According to thepresent example, the computer system includes a bus 201, at least oneprocessor 202, at least one communication port 203, a main memory 24, aremovable storage media 205, a read only memory 206, and a mass storage207.

Processor(s) 202 can be any known processor, such as, but not limitedto, an Intel® Itanium® or Itanium 2® processor(s), or AMD® Opteron® orAthlon MP® processor(s), or Motorola® lines of processors. Communicationport(s) 203 can be any of an RS-232 port for use with a modem baseddialup connection, a 10/100 Ethernet port, or a Gigabit port usingcopper or fiber, for example. Communication port(s) 203 may be chosendepending on a network such a Local Area Network (LAN), Wide AreaNetwork (WAN), or any network to which the computer system 200 connects.Main memory 204 can be Random Access Memory (RAM), or any other dynamicstorage device(s) commonly known to one of ordinary skill in the art.Read only memory 206 can be any static storage device(s) such asProgrammable Read Only Memory (PROM) chips for storing staticinformation such as instructions for processor 202, for example.

Mass storage 207 can be used to store information and instructions. Forexample, hard disks such as the Adaptec® family of SCSI drives, anoptical disc, an array of disks such as RAID (e.g. the Adaptec family ofRAID drives), or any other mass storage devices may be used, forexample. Bus 201 communicably couples processor(s) 202 with the othermemory, storage and communication blocks. Bus 201 can be a PCI/PCI-X orSCSI based system bus depending on the storage devices used, forexample. Removable storage media 205 can be any kind of externalhard-drives, floppy drives, flash drives, IOMEGA® Zip Drives, CompactDisc-Read Only Memory (CD-ROM), Compact Disc-Re-Writable (CD-RW), orDigital Video Disk-Read Only Memory (DVD-ROM), for example. Thecomponents described above are meant to exemplify some types ofpossibilities. In no way should the aforementioned examples limit thescope of the invention, as they are only exemplary embodiments.

FIG. 3 illustrates an emergency response environment with a system 300that monitors three-dimensional interaction, according to embodiments ofthe present invention. System 300 includes a sensor or sensor array 1.Sensor 1 may be a camera, video camera, or other imaging device capableof collecting visual information. According to some embodiments of thepresent invention, sensor 1 is a sensor array that includes an imagecapture device, for example a color image capture device, as well as adepth determining device, for example an infrared emitter and infrareddepth sensor. Sensor 1 may also include an audio capture device. Forexample, sensor 1 may be a sensor array such as a Kinect® sensor arrayavailable from Microsoft Corporation. Sensor 1 may also or alternativelybe a LEAP™ device available from Leap Motion, Inc. Sensor 1 may be, orinclude, a wide variety of hardware that permits collection of visual,depth, audio, and color information and the like, according toembodiments of the present invention.

Sensor 1 may be placed within an emergency response environment, forexample in the back 152 of an ambulance 101, such that activities of thepatient 116 and/or crew members 2, 3 are at least partially within itsfield of view. For example, sensor 1 may be mounted on a wall or ceilingof the back compartment 152 of the ambulance 101. The sensor 1 may alsoinclude, within its field of view, a patient support 4, such as a bed,cot, or stretcher, upon which a patient 116 is laying and/or beingtreated. The back 152 of the ambulance 101 may further include a supplycabinet 5, for example a medicine cabinet or narcotics cabinet, whichmay be stocked with medicines, for example narcotic 6.

FIG. 4 illustrates a system including a vehicle control system 104communicably coupled with a sensor array 1, according to embodiments ofthe present invention. Sensor array 1 may include an imaging device 9, adepth sensor system 10, and/or an audio input 11, according toembodiments of the present invention. VCS 104 may also be communicablycoupled with a patient monitoring device 106, a charting system 108, anavigation system 110, and vehicle operations systems 8. The vehicleoperation systems 8 may include sensors and controllers installed in thevehicle relating to vehicle safety and/or operation, including bothmanufacturer-installed and aftermarket devices, for example vehiclespeed sensors, seatbelt detectors, accelerometers, and other vehicle-and safety-related devices, including without limitation those describedin U.S. Provisional Patent Application Ser. No. 61/656,527, filed onJun. 7, 2012, which is incorporated by reference herein in its entiretyfor all purposes.

Vehicle control system 104 may be configured to create, maintain, and/orupdate an encounter record 7, which may be stored locally in anemergency response environment (for example in database 118) and/orremotely on an enterprise database 130. The encounter record 7 mayinclude information obtained by the vehicle control system 104 and eachof the devices to which VCS 104 is communicably coupled. Records in theencounter record 7 may be specific to an encounter with a particularpatient 116, and/or a particular dispatch of the vehicle 101, forexample.

The VCS 104 may be configured to track interactions in the emergencyresponse environment, for example interactions by and among caregivers2, 3, and patient 4 and/or objects in the emergency responseenvironment. The VCS 104 may be configured to receive color images anddepth information from within a field of view of the sensor array 1. TheVCS 104 may also be configured to maintain an emergency encounter record7, either locally and/or remotely. The VCS 104 monitors a position of anobject and/or movement of the object in the emergency responseenvironment based on the color images and depth information received bythe sensor array 1. For example, the sensor array 1 may be a Kinect®sensor array, and the VCS 104 may include software that receives datafrom the sensor array 1 to detect or approximate movements and locationsof human bodies and their respective linkages (skeletal joints andbones) in three-dimensional space.

As such, the VCS 104 can distinguish between different humans in thefield of view of the sensor 1, and can monitor or observe the movementsof two or more of such humans in the field of view. According to someembodiments of the present invention, the VCS 104 is configured torecognize which of the humans is a patient and which is a caregiver. Forexample, VCS 104 may recognize a human as a patient by observing thatthe particular human is laying relatively still on the patient support4, while another human is an EMS technician 2 because the other human isstanding up or moving around the back of the ambulance 101. The VCS 104may be configured to track or monitor three-dimensional movements of oneor more humans in the emergency response environment by approximatingelements of their basic skeletal structure and, as such, can determinewhen two humans are in contact or close proximity. For example, the VCS104 can determine when a hand or arm of the EMS technician 2 reachesover and touches an area of the patient's 116 body, according toembodiments of the present invention.

Any or all of the information received by the VCS 104 from the sensorarray 1, as well as any additional data or information derived from suchsensor information, may be stored to the encounter record 7. Suchinformation may also be stored to the encounter record 7 in a mannerthat correlates it with other data in the encounter record 7 from otherdevices, for example records in the encounter record 7 may include atime index and/or a patient identification.

According to embodiments of the present invention, the VCS 104 isconfigured to record into the emergency encounter record 7 an occurrenceof a condition. Such condition may be based on the position of theobject and/or the movement of the object. For example, the object may bea human, and the VCS 104 may monitor the human's movement (or a skeletalapproximation thereof) in three-dimensional space, and make an entry inthe encounter record 7 when the human or part of the human intersects acertain location (e.g. within the ambulance 101), or remains in aparticular location for a certain amount of time, or intersects or nearsanother object. The VCS 104 may be configured to make an entry to theencounter record 7 when one object (e.g. a human) comes within a certaindistance of another object (e.g. another human), for example a zero orminimal distance at which the first object is touching the secondobject. As such, the VCS 104 may be configured to mark the encounterrecord 7 when a caregiver 2 or 3 approached the patient 116 and/ortouched the patient 116, or when an object approached or touched thepatient 116.

The VCS 104 may be configured to update the encounter record 7 invarious ways based on the observance of a condition based onthree-dimensional visual and position data. For example, the VCS 104 maybe configured to enter into the encounter record 7 a time at which thecondition occurred, and/or an identification of the condition or type ofcondition that occurred, and/or other data coinciding with theoccurrence of the condition, for example video data or color imagescovering the time or time range when the condition occurred. In somecases, the VCS 104 receives streaming clinical data about a patient 116,for example from a defibrillator or other patient monitoring device 106communicably coupled to the patient, and correlates at least a portionof the streaming clinical data in the emergency encounter record 7 withthe occurrence of the condition based on the sensor's 1 visual data.According to embodiments of the present invention, correlating some orall of the streaming clinical data includes flagging some or all of thestreaming clinical data that corresponds to a time of the occurrence ofthe condition.

FIG. 7 illustrates a flow chart 700 showing the recording of anoccurrence of a condition based on three-dimensional position and shapevisual data, according to embodiments of the present invention. One ormore distinct objects are identified (block 702), for example by VCS 104and sensor 1. The position and/or movement of the one or more objectsare tracked or otherwise monitored or modeled (block 704), and based onsuch tracking the VCS 104 identifies the occurrence of a condition(block 706). The occurrence of the condition, or information about thecondition, is recorded in the patient encounter record 7 (block 708).

FIG. 8 illustrates a flow chart 800 describing a similar method ingreater detail, according to embodiments of the present invention. Anindividual human or distinct humans are identified in an emergencyresponse environment, for example the back of an ambulance (block 802).At least one of the humans is identified as a patient (block 804). Theposition and/or movement of the one or more humans is observed ortracked or otherwise modeled (block 806), and based thereon the VCS 104identifies the occurrence of a condition, for example the occurrence ofpatient treatment (block 808). Information about the patient contact maybe recorded in the encounter record 7 (block 810), for example byrecording a time or time range at which the condition (e.g. treatment)occurred (block 812), and/or by recording a type of contact (e.g.treatment) which occurred (block 814).

For example, if the sensor 1 data supplied to the VCS 104 wasinterpreted by the VCS 104 as a caregiver's 2 hand going to the head ormouth area of the patient 116, the VCS 104 may update the encounterrecord 7 to reflect that an oral medication was or may have beenadministered to the patient 116, and the particular time which thisoccurred. Alternatively, or in addition, the VCS 104 may be configuredto prompt the EMS technician 2 or other caregiver at a later time, forexample after the emergency encounter or at the end of a standard shift,to confirm or validate the perceived interactions or conditions thatwere entered into the patient encounter record 7. For example, the VCS104 might observe the occurrence of the EMS technician's 2 hand going tothe face of the patient 116 and flag such occurrence as the possibleadministration of an oral medication, but when prompting the EMStechnician 2 for later confirmation, may give the EMS technician 2 theability to edit the observation to reflect that the interaction wasinstead a turning of the head of the patient, or some other reason forwhy the caregiver 2 contacted the patient 116.

FIG. 9 depicts a flow chart 900 illustrating a method for monitoringthree-dimensional interaction in an emergency response environment forinventory control, according to embodiments of the present invention.The VCS 104 may identify a particular location within the emergencyresponse environment, for example a supply cabinet 5, using sensor 1 andknown information about the environment (block 902). The VCS 104 mayalso be configured for customization regarding the locations of certainitems in the emergency response environment. For example, during aninitialization and/or configuration protocol, the VCS 104 may prompt theuser to run the user's finger or hand around an outer perimeter of asupply cabinet 5 and/or a door thereto, so that the VCS 104 can log thethree-dimensional position of the supply cabinet 5. Such cabinet 5 maybe, for example, a narcotics cabinet 5 to which access is oftencontrolled for safety and security reasons.

The VCS 104 may identify individual humans in the emergency responseenvironment, for example the back of an ambulance (block 904), and trackthe position and/or movement of such humans (block 906). This may bedone with visual and depth information received from the sensor array 1,according to embodiments of the present invention. Based on such visualand depth information received from the sensor array 1, the VCS 104 mayalso detect or track three-dimensional movement of an object in theemergency response environment, for example a non-human object. The VCS104 may determine an occurrence of contact between the human body andthe object (block 908), for example an occurrence of the human body or aportion thereof approaching and/or intersecting the narcotics cabinet 5.The VCS 104 may also record an entry in an emergency encounter record 7based on the occurrence of the contact, for example a note that thecabinet 5 was accessed (block 910) along with a time (block 912) and/oran identity of the person who accessed the cabinet 5 (block 914). TheVCS 104 may be configured to observe the occurrence of various differenttypes of conditions of note. For example, the VCS 104 may be configuredto detect an intersection of a human form with the area of the door oropening to the cabinet 5. The VCS 104 may be configured to detect that ashape that correlates to the shape of the narcotic medication 6 has gonefrom inside such area of the door or cabinet opening to outside sucharea. VCS 104 may also be configured to note whether a human has anobject in the human's hand as well as the shape and/or size of theobject. The VCS 104 may further be configured to update an inventorydatabase, based on the occurrence of the removal, to reflect that thenarcotic medication has been used and needs restocking. Similarprocesses may be used to track the use of other objects and theinventory associated therewith, as well as to track in general theintersection of objects with humans and use thereby, according toembodiments of the present invention. According to some embodiments ofthe present invention, the occurrence of an access event to theparticular cabinet 5 may further trigger other information gathering,for example it may trigger a camera on the inside of the cabinet 5and/or another video camera elsewhere in the vehicle 101. The identityof each crew member accessing the cabinet 5 may be recorded in theencounter record 7, according to embodiments of the present invention.

FIG. 10 depicts a flow chart 1000 illustrating a method for gesturerecognition in an emergency response environment, according toembodiments of the present invention. While system 400, including VCS104 and sensors 1, may be configured to track motions, positions, andinteractions of humans and objects as described above, system 400 aswell as VCS 104 and sensors 1 may also or alternatively be configured tomonitor such visual information for the occurrence of gestures. In somecases, three-dimensional position and visual information may be used tomonitor for gestures; in other cases, mere visual information may beused to detect gestures (e.g. based on pattern recognition or othervisual cues or patterns). As such, sensor 1 may be one of a number ofvarious types of sensors or sensor arrays.

VCS 104 may be configured to track an entire human body and/or one ormore portions thereof to identify gestures being made, for examplegestures being made by one or more hands and/or fingers or by the headand/or neck (block 1002). VCS 104 receives visual information about atleast a portion of a human body from at least one sensor 1, andmaintains the encounter record 7. The VCS 104 is configured to monitorthe visual information to determine movements of the at least theportion of the human body (for example the hand or the head), and torecognize an occurrence of a gesture based on the movements of the atleast the portion of the human body. For example, the VCS 104 recognizesone or more hand or finger gestures based on visual and/or depthinformation received by sensor 1, for example one or more hand or fingergestures listed in FIG. 5. The VCS 104 may also recognize one or morehead or facial gestures based on visual and/or depth informationreceived by sensor 1, for example one or more head or facial gestureslisted in FIG. 6.

Examples of hand or finger gestures may include waving a hand or finger,making a fist, raising the fist, shaking the first, making the “thumbsup” signal, spreading fingers apart, displaying a count (e.g. zero, one,two, three, four, five, six, seven, eight, nine, or ten digitsextended), pointing, moving hands together, pulling hands apart, and/ortapping on the wrist. Examples of head or facial gestures may includenodding the head, bobbing the head, shaking the head side-to-side as inthe “no” gesture, shaking the head up and down as in the “yes” gesture,blinking, opening or closing the mouth, sticking the tongue out, raisingor lowering eyebrows, and/or opening or closing the eyes.

When the VCS 104 recognizes a gesture, the VCS 104 records an entry inthe emergency encounter record 7 based on the occurrence of the gesture(block 1004). Such a gesture may be artificial, or alternatively such agesture may be natural. An artificial gesture is a gesture made by ahuman for the primary purpose of triggering the condition with VCS 104.As such, an artificial gesture may be a gesture that would not normallybe made in the normal course of treating a patient 116 in an emergencyresponse environment. For example, making a “thumbs up” signal is oneexample of an artificial gesture. A patient whose head is involuntarilybobbing is an example of a natural gesture, or a gesture which is notperformed only to trigger VCS 104.

The entry which the VCS 104 makes in the patient encounter record 7based on the recognition of the gesture may include information aboutthe type of gesture made (block 1006), information about the time atwhich the gesture was made (block 1008), and/or information about otherdata values at the time the gesture was made (block 1010), for exampleinformation about the crew (block 1012), patient clinical data (block1014), and vehicle operation or safety conditions (block 1016). Forexample, VCS 104 may be configured to write the patient's 116 currentblood pressure reading to the encounter record 7 whenever VCS 104receives visual and/or depth information from the sensor 1 indicatingthat the caregiver 2 attending to the patient 116 taps his or her leftwrist with the right hand or fingers (tapping the location where a watchwould normally be worn). Successive gestures may be used to take the VCS104 down various pathways and/or treatment protocols, or to confirmprevious gestures or options that become available because of thosegestures. For example, the VCS 104 may be configured to record a bloodpressure reading to the encounter record 7 when it identifies the wristtapping gesture followed by a chest tapping gesture, and may beconfigured to record an ECG waveform signal to the encounter record 7when it identifies the same wrist tapping gesture followed by aback-of-the-neck tapping gesture. The VCS 104 may also be configured torecord in the encounter record 7 the audiovisual (e.g. video and/oraudio) information received during or within a certain time range of thegesture, according to embodiments of the present invention.

According to some embodiments of the present invention, the VCS isconfigured to identify simultaneous occurrence of gestures, for exampletwo or more gestures selected from FIG. 5, FIG. 6, or any other naturalor artificial gestures. According to some embodiments of the presentinvention, the VCS 104 is configured to identify simultaneous occurrenceof gestures along with position and/or movement information for entirehuman bodies or portions thereof, or simultaneous occurrence of otherfactors such as vehicle position along the ambulance route, patientvital signs, and/or vehicle speed. VCS 104 may also be configured toidentify simultaneous occurrence of gestures by the same person, forexample a different or similar gesture with each hand, or a hand and ahead. For example, the VCS 104 may be configured to recognize a handwaiving gesture and to make a record in the encounter record 7 andnotify the ambulance driver to slow down if the hand waiving gesture isreceived at a time when the vehicle speed is exceeding 60 miles perhour. In this way, the visually recognized gestures may be paired orcorrelated or combined with other information received by VCS 104,either in the creation of the condition which triggers a further event(such as writing to the encounter record 7 or creating a notification orsome other action), or in the creation of the entry to the encounterrecord 7 itself (for example the types of information that would beflagged or gathered or otherwise noted upon occurrence of thecondition).

According to some embodiments, the VCS 104 identifies (either in theencounter record 7 or for other devices) whether a patient is beingtransported by the vehicle 101, for example by determining whether ahuman figure is sitting on or laying on the patient support 4. The VCS104 may also identify the position of a patient or a crew member, forexample whether the patient or crew member is sitting or standing. TheVCS 104 may also receive from sensor 1 information about structuresbeyond a normal emergency response environment, for example larger-scaledepth images of emergency incidents such as buildings on fire, to aid inthe location of emergency workers and/or victims.

Although one sensor 1 is shown and described, multiple sensors 1, eitherof the same type of different types, may be communicably coupled withVCS 104. Multiple sensors 1 may be used to expand the field or depth ofview, or to collect similar information from a different viewing angle,in order to observe more objects or humans, or gather more detailedinformation about shapes and/or movements. And although sensor 1 isdescribed as being mounted within a vehicle, sensor 1 or multiplesthereof may alternatively be mounted on a device (for example adefibrillator taken to an emergency response scene) and/or on a person(for example on a crew member's helmet).

Embodiments of the present invention may also be used for chartingand/or counting functions. Often, medics must reconstruct past eventsthat occurred during patient treatment. Embodiments of the presentinvention improve accuracy and help to accurately document times atwhich various events occurred. For example, the VCS 104 may recognizeboundaries of multiple cabinets or storage areas within an ambulance101, and may log the times at which each storage area was accessed by amedic, as well as the identity (e.g. obtained from voice or body orfacial recognition) of the medic who accessed the area. Such a “boundingvolume” may be preprogrammed into VCS 104 and/or customized orinitialized upon installation of VCS 104, sensor 1, and/or a new storagearea. The VCS 104 may count a number of boxes on the floor of theambulance to determine a number of items used in the encounter, andreconcile that with the medications and other durable goods charted forthe patient encounter. The VCS 104 may then prompt the medic foradditional information to help reconcile the encounter record 7.

As described above, the system 400 may also determine when a patient isbeing touched, either by another human or by an implement held byanother human. This information may be used either during the patientencounter, or afterward, to determine whether inappropriate patientcontact has occurred. The system 400 may determine when an IV is beingstarted. System 400 may also use gesture-based charting, for examplequick-logging with artificial gestures, to save time over manual entryor typing of such information. Embodiments of the present invention mayalso include voice recognition, which may filter out siren sounds orroad sounds, and which may also provide feedback to the crew.Embodiments of the present invention may also be configured to identifycrew members, for example through facial recognition, patternrecognition, name badge reading, skeletal modeling, habits or movements,or via another mechanism such as crew logins or RFID badges which arealso communicably coupled to VCS 104. According to some embodiments ofthe present invention, the system 400 may be used for securitymonitoring, to detect the presence of unidentified or unwanted intrudersin the vehicle 101.

According to some embodiments of the present invention, the system 400may be used to begin tracking a person when the person makes a gestureor performs a certain activity, and then continue to track the sameperson after the gesture or activity, for a certain period of time oruntil another event occurs, for example another visual event. In someembodiments, the system 400 identifies an operator of a medical deviceusing visual information; for example, a patient monitoring device 106,such as a defibrillator, may include a camera or other type of sensorarray 1, and upon use of the device 106 the device 106 may observevisual characteristics of the person directly in front of the device 106in order to identify the person or monitor or interpret activities ofthat person. The system 400 may also be configured to recognize oridentify in its field of view equipment used by medical personnel,either by visual cues or otherwise, and may perform similar medicalpersonnel identification or visual monitoring even when the camera orsensor array 1 is not in or near the device being used. Such multipledevices used by medical personnel may be wirelessly or otherwisecommunicably coupled with each other and/or with system 400, so thatactivities performed on various devices and by the personnel arecorrelated for a more complete patient record without requiring manualannotation, according to embodiments of the present invention. Thesystem 400 may be mounted not only in a vehicle, such as the back of anambulance, but system 400 and/or parts thereof may also be integratedinto or mounted on a medical device, including a portable medical devicesuch as a defibrillator.

The system 400 may also be configured to “remember” a person based onthat person's gestures; for example, the system 400 may observe certaingestures performed by a person one day after the person identifieshimself or herself to the system 400, and may then visually identify thesame person the next day based on observing similar gestures, even ifthe person has not specifically identified himself or herself to thesystem 400 on the following occasion. The system 400 may also beconfigured to count the number of distinct individual people in a givenarea, according to embodiments of the present invention.

The system 400 may also be configured to monitor certain activities andto interpret various aspects of those activities, and even to providefeedback to the performer of the activities either in real time or in alater review. For example, the system 400 may monitor an EMStechnician's twelve-lead placement on a patient, and/or may provideadaptive feedback, for example adaptive feedback to a person who isadministering cardiopulmonary resuscitation. The system 400 may also beconfigured to identify a certain portion of the body, or an object heldby a person, and to track the movement of the body part or object andrecord the tracked motion as writing. For example, an EMS techniciancould write numbers, letters, or words in the air using a finger, andthe system 400 may be configured to record such movement as writing. TheEMS technician may initiate such “air writing” recording mode with agesture or other activation; in other embodiments, the system 400automatically recognizes such “air writing” based on the absence ofother objects with which the user's hand or finger could be interacting,for example for a certain period of time. Such recording capabilitiesmay save the EMS technician time in data entry or patient charting, andwould permit the medical professional to create charting entries andother writings even when the medical professional's hands are dirty, orwhen the medical professional does not wish to physically touch devicesso as to maintain sterility for hands or gloved hands, according toembodiments of the present invention.

Various modifications and additions can be made to the exemplaryembodiments discussed without departing from the scope of the presentinvention. For example, while the embodiments described above refer toparticular features, the scope of this invention also includesembodiments having different combinations of features and embodimentsthat do not include all of the described features. Accordingly, thescope of the present invention is intended to embrace all suchalternatives, modifications, and variations as fall within the scope ofthe claims, together with all equivalents thereof.

What is claimed is:
 1. A method for tracking interactions in anemergency response environment, the method comprising: receiving colorimages and depth information from within a field of view of a sensorarray; maintaining an emergency encounter record; monitoring one or bothof a position of an object and movement of the object in the emergencyresponse environment based on the color images and depth informationreceived by the sensor array; and recording an occurrence of a conditionin the emergency encounter record, wherein the condition is based on theone or both of the position of the object and the movement of theobject.
 2. The method of claim 1, wherein the object is a human, andwherein monitoring one or both of the position of the object andmovement of the object comprises monitoring one or both of the positionof the human and movement of an at least partial skeletal approximationof the human.
 3. The method of claim 2, wherein the human is a firstobject, wherein the condition comprises the at least partial skeletalapproximation of the human coming within a certain distance of a secondobject.
 4. The method of claim 3, wherein the human is a first human,and wherein the second object is a second human.
 5. The method of claim4, wherein the condition comprises the first human touching the secondhuman.
 6. The method of claim 5, wherein the second human is a patientbeing treated by the first human in the emergency response environment.7. The method of claim 1, wherein recording the occurrence of thecondition comprises recording a time at which the condition occurs. 8.The method of claim 7, wherein recording the occurrence of the conditionfurther comprises recording a type of the condition.
 9. The method ofclaim 7, wherein recording the occurrence of the condition furthercomprises recording as video footage the color images received duringthe occurrence of the condition.
 10. The method of claim 1, furthercomprising: receiving streaming clinical data about a patient, andcorrelating at least a portion of the streaming clinical data in theemergency encounter record with the occurrence of the condition.
 11. Themethod of claim 10, wherein correlating the at least the portion of thestreaming clinical data comprises flagging the at least the portion ofthe streaming clinical data corresponding to a time of the occurrence ofthe condition.
 12. A system for tracking interactions in an emergencyresponse environment, the system comprising: a sensor array, wherein thesensor array is adapted to receive color images and depth information inits field of view; a control system communicably coupled to the sensorarray, the control system configured to: maintain an emergency encounterrecord; monitor one or both of position and movement of an object in theemergency response environment based on the color images and depthinformation received from the sensor array; and record an occurrence ofa condition in the emergency encounter record, wherein the condition isbased on the one or both of position and movement of the object.
 13. Amethod for inventory control in an emergency response environment, themethod comprising: detecting three-dimensional movement of a human bodyin the emergency response environment with a sensor array, wherein thesensor array generates visual information and depth information aboutthe emergency response environment; detecting three-dimensional movementof an object in the emergency response environment; determining anoccurrence of contact between the human body and the object; andrecording an entry in an emergency encounter record based on theoccurrence of the contact.
 14. The method of claim 13, wherein theobject is a narcotic medication stored in an enclosure in the emergencyresponse environment, the method further comprising: determining, basedon the detection of the three-dimensional movement of the human body andthe object, an occurrence of intersection of the human body with theenclosure; and recording an entry in the emergency encounter recordbased on the occurrence of the intersection.
 15. The method of claim 13,wherein the object is a narcotic medication stored in an enclosure inthe emergency response environment, the method further comprising:determining, based on the detection of the three-dimensional movement ofthe object, an occurrence of removal of the narcotic medication from theenclosure; and recording an entry in the emergency encounter recordbased on the occurrence of the removal.
 16. The method of claim 15,further comprising updating an inventory database, based on theoccurrence of the removal, to reflect that the narcotic medication hasbeen used and needs restocking.